1. Field of the Invention
This invention relates generally to computer testing, and more particularly to comparing XML based test reports.
2. Description of the Related Art
Software applications typically undergo constant review and improvement, which verifies the functions of the software and updates the software to incorporate new features and functions. However, each new feature and error fix may cause addition errors. As such, test engineers generally apply a series of test cases, known as a test suite, to the application to test each new update and error fix. A typical test suite can include thousands of individual test cases.
For example, in a Java™ application, a test case generally includes an assertion, such as pushing a button, followed by an occurrence of a particular event. In addition, the test case validates the occurrence of the event, verifying whether the event occurs correctly. For example, a test case can verify the function of a “tone” button on an Internet web page application. When the tone button is pressed, a tone should be produced. The test case simulates a computer mouse dragging a cursor to the button, pressing the button, and then releasing the button. The test case then validates whether the tone was correctly produced. Finally, a test result is output to a test report, which includes at least one test result for each one of the test cases.
Often, test reports are output in a text format so a software test engineer can review the test results in a written form. However, unformatted text-based test reports can be difficult for a human engineer to read. To make test reports easier to read, test suites often output the test reports in a Hypertext Markup Language (HTML) format, rather than an unformatted text format. HTML format allows a user to view the data using a browser application, which renders the data easy for a human to read.
Conventional test applications typically use an HTML reporter to convert test results into HTML format. HTML surrounds data with tags that define how a browser should present the data to a user. Hence, an HTML test report file represents a view of the test result data, rather than the data itself. As such, an HTML test report file cannot easily be displayed in different views without severely altering the tags and other content of the HTML file. Thus, if a specific view of the test result data is needed, the new data view cannot be easily created without severely altering the HTML test report file.
Different users often require different views of the test report data. For example, a department manager may require a listing of test regressions, while an application designer may require a listing of all test failures. Using an HTML based test report file, these two data views generally cannot be easily created using the same HTML test report file. Creating these views typically requires the HTML test report file to be examined and manually altered to create new HTML files, which include only the information requested.
A test regression is when a test case fails in a first revision of the software (i.e., revision 1.0), the failure is corrected in a subsequent revision (i.e., revision 1.01) and then the same failure reoccurs in yet another subsequent revision (i.e., revision 2.0). A test regression can identify improvements or error fixes that caused problems that were previously fixed. Properly identifying a test regression can assist the software engineer to quickly identify the cause of the test regression. Unfortunately, a test engineer must access the historical knowledge base of all previous test results to accurately identify test regressions.
As test suites grow ever larger, the task of interpreting the information in the test report becomes more and more labor intensive. A typical test suite might include two thousand or more individual test cases. The test report can therefore include two thousand or more entries. Applying the test suite to a revision 1.01 of a software application outputs a revision 1.01 test report. The test engineer can then compare each of the approximately two thousand entries in the revision 1.01 test report to the corresponding entry in a previous revision (e.g., revision 1.0) test report. Manually comparing two or more test reports can be very labor intensive, often requiring many hours to review just one new test report. However, in the fast-paced, limited budget environment of a typical software development project, the number of hours available to evaluate a test report is limited.
Typically the test engineer desires to identify the differences between the revision 1.0 test report and the revision 1.01 test report. By identifying the differences, the test engineer can then track progress toward resolving software failures. The differences typically include any tests that have failed in revision 1.0 test but did not fail in revision 1.01. In addition, the differences may also include any tests that failed in revision 1.01 but did not fail in revision 1.0 test.
Comparing two test reports can be further complicated when the test suite is modified to include additional test cases. In such an instance, the test report comparison must first identify the common test cases and then only compare the results of the common test cases. Then the test cases that are not common must be individually examined to determine if the test cases failed or passed.
One approach to making comparison and analysis of test reports easier is to store the test reports in a database environment. However, storing the test reports in a database requires a database manager and a database administrator to function properly. A database manager and administrator can also complicate and delay the test result analysis as the database can only be viewed and accessed through the database manager.
In view of the foregoing, there is a need for techniques that compare data from test reports that is more user-friendly and requires fewer man-hours and less actual elapsed time.